Photo-Based User Interfaces: Picture It, Tag It, Use It

Pervasive environments can be hard to configure and interact with using handheld computing devices, due to the mismatch between physical and digital worlds. Usually, smart resources in the user’s vicinity are discovered and presented in a menu on the user’s device from where they can be accessed. However, in environments with many embedded resources it becomes hard to identify resources by means of a textual description and to get aware of the tasks they support. As an alternative to menu-driven interfaces, we propose annotated photos as a means for controlling a pervasive environment. We present a tool that enables people to picture their own environment and use photos as building blocks to create an interactive digital view on their surroundings. To demonstrate and evaluate our approach, we engineered a pervasive prototype application that is operated through a photo-based user interface and assembled using ontologies.